Method of processing parallax information comprised in a signal

ABSTRACT

This invention relates to a method and a device of processing parallax information comprised in a signal. A signal comprising parallax map associated with further image information is received. A first data is obtained from the signal indicative of first parallax map constraints. A second data is obtained from the signal indicative of second parallax map constraints. Third data matching third parallax map constraints of a target device is determined by means of processing the at least the first data and the second data. This third data is used to generate an updated signal matching the parallax map information constraints of the target device.

FIELD OF THE INVENTION

The present invention relates to a method and a device for processingparallax information comprised in a signal.

BACKGROUND OF THE INVENTION

Three-dimensional (3D) display devices add a third dimension (depth) tothe viewing experience by providing each of the viewer's eyes withdifferent views of the scene that is being watched. Many 3D displaydevices use stereo input, which means that two different but relatedviews are provided. This is used, for example, in standard 3D cinema(where glasses are used to separate left and right views for theviewer's eyes). Instead of, for example 50 frames (of image data) asecond being provided, in a stereo system 100 frames a second areprovided, being 50 for the left eye, and 50 for the right eye. Eachframe of a pair comprises a slightly different view of the same scene,which the brain combines to create a three-dimensional image. As aresult of the adoption of this technology in 3D cinemas, there is a lotof stereo content available. It is also possible that there are homecinema enthusiasts who will want to replicate the cinema experience athome and build or install stereo projection systems.

However, the use of glasses that are associated with stereo 3D systemsis cumbersome for many applications, such as 3D signage and also morecasual home 3DTV viewing. Glasses-free systems (also calledauto-stereoscopic systems) often provide more than two views of thescene to provide freedom of movement of the viewer, and since the numberof views varies, the representation that is often used in theseapplications is the image+depth format, where one image and its depthmap provide the information required for rendering as many views asneeded.

A problem that exists with systems that provide parallax information isthat the structure of the parallax information (which is additional tothe image data), will be optimized for a particular target renderingsystem or device. For example, if a depth map is provided, then this maybe designed with a particular target system in mind. For example, it maybe assumed in the creation of the map that the end system is designed toprovide 6 different views (the user will only ever see two of the sixviews, depending upon their position). The choice of 6 views may bebased upon what is perceived to be the most likely (or average)configuration of the end system. However the parallax informationcontained within the signal may not be appropriate for the renderingthat will occur at the display device

BRIEF DESCRIPTION OF THE INVENTION

The object of the present invention is to improve upon the known art.

According to one aspect the present invention relates to a method ofprocessing parallax information comprised in a signal, comprising:

receiving a signal comprising at least one parallax map associated withfurther image information,

obtaining first data from the signal indicative of first parallax mapconstraints,

obtaining a second data from the signal indicative of second parallaxmap constraints,

determining third data matching third parallax map constraints of atarget device by means of processing at least the first data and thesecond data, the third data being adapted to generate an updated signalmatching the parallax map information constraints of the target device.

It is thus possible, in case that neither the first nor the second datamatch the constraints of the target device, to use the data as input ingenerating an updated third data that matches the target device. Theimage information may be a still image, or a frame or field of avideo-sequence.

In one embodiment, the first data and the second data are parallaxinformation transforms, the first and the second parallax mapconstraints being first and second depth ranges for the imageinformation, the third data being a third parallax information transformand the third parallax map constraints being a third depth range.

With the term with parallax information is meant depth-relatedinformation or disparity-related information or a combination of both.Here depth-related information is used for information that representsan indication as to the apparent distance of image information to theviewer/camera. In turn, disparity-related information is used forinformation that represents an indication as to the apparent shift ofimage elements between views, i.e. the displacement in images for theleft eye and right eye.

In one embodiment, the first parallax information transform is anidentity information transform of the received parallax map.

In one embodiment, the second parallax information transform is parallaxinformation transform obtained by using the identity informationtransform as input which is processed, the processing resulting inoutputting the second parallax information transform.

In one embodiment, the signal is a video signal and where the secondparallax information transform is comprised in the video signal asmetadata.

In one embodiment, the metadata comprises at least one of: a mappingfunction related to the parallax information,

an inverse of a mapping function related to the parallax information,and

a coefficient for a mapping function related to the parallaxinformation.

In some cases it might be necessary or possible to determine for exampledisparity information from cinema stereo, and then add a mapping totransform those disparities to a format more suitable for areduced-depth-range home 3D display. The disparity for the latter (theresult of applying the transform) is then only first generated at thereceiving end in the home. This meta-data is a “backwards” mapping toundo a mapping that had been done at the content creation side.

Accordingly, the at least second parallax information transform may beconsidered as adjusted parallax information suitable to a specificend-receiver. The principle of the metadata is that it makes it possibleto obtain data which could otherwise not be obtained from the (original)parallax information without the metadata. As an example, first parallaxinformation is sent to a 3D display device. The metadata relates to thegeneration of the parallax information, i.e. the methology of how theywere obtained (e.g. via a function or a look-up table and the like). Themetadata allows the receiver to work back from parallax information tothe underlying data that was used to create the parallax information orto new parallax information better suited to a specific target device.The result is that said second parallax information transform iscreated, i.e. parallax information that is adjusted to a 3D display atthe receiver side.

In one embodiment, the step of determining third data matching the thirdparallax map constraints of the target device comprises interpolatingbetween two respective parallax information transforms from a set ofparallax information transforms, the set of parallax informationtransforms comprising the first and second parallax informationtransforms, the third data being a third parallax information transformmatching the depth range of the target device.

Ideally, a 3D display can show a large parallax range. This howeverrequires several viewing conditions to be met, e.g. the screen must belarge, the screen must be watched from a large distance and that theseparation between views must be very good. These viewing conditions arenot always met. Therefore, this “original depth” gives rise to saidfirst parallax range, whereas said second parallax information transformresults in a second depth signal with said second parallax range. Theadvantage with this embodiment is that when the parallax range of a 3Ddisplay device does not match either of these ranges, a novel transformcan be computed from the two (or more) transforms by e.g. interpolation.In that way, the depth range of the signal can precisely be tuned to theavailable parallax range of a 3D display, thereby enabling improved 3Drendering.

In one embodiment, the set of parallax information transforms furthercomprises a further parallax information transform based on further datafrom the signal.

In one embodiment, the parallax information transforms used as input indetermining the updated parallax information transform are selectedbased on a selection rule. In one embodiment, the selection rule definesselecting parallax information transforms that fall within apre-determined depth range of the target device. This predeterminedrange could e.g. be the closest depth range magnitude.

In one embodiment, the target device is a 3 dimensional (3D) displaysystem and where the respective parallax map constraints comprise atleast one from:

the parallax or depth range of the 3D display device,

the display distance between a viewer and the 3D display device, and

a location parameter indicating the position of the viewer from the 3Ddisplay device.

In one embodiment, the updated signal is subsequently forwarded to thetarget device where the updated signal is used to adjust the parallaxmap so as to render image elements for view information for athree-dimensional image that falls within the available parallax rangeof the target device. Accordingly, the processing of the parallaxinformation may be performed externally from e.g. a three-dimensional(3D) display device.

According to another aspect, the present invention relates to a computerprogram product for instructing a processing unit to execute the abovemethod steps when the product is run on a computer.

According to still another aspect, the present invention relates to adevice for processing parallax information comprised in a signal,comprising:

a receiver for receiving a signal comprising at least one parallax mapassociated to image information,

a processor for

obtaining first data from the signal indicative of first parallax mapconstraints,

obtaining second data from the signal indicative of second parallax mapconstraints and

determining third data matching third parallax map constraints of atarget device by means of processing at least the first and the seconddata, the third data being adapted to generate an updated signalmatching the parallax map information constraints of the target device.

Accordingly, a device is provided that is capable of, in case thatneither the first nor the second data match the constraints of thetarget device, to use the data as input in generating an updated thirddata that matches the target device. The image information may be stillimage, or a frame or field of a video-sequence.

The device may be an integral part of a set-top box, Blu-ray Discplayer, a 3D display device, a stereo display, a PC computer device, ora portable computer device.

According to still another aspect, the present invention relates to athree-dimensional (3D) display device comprising said device.

In one embodiment, the 3D display device is an auto-stereoscopic displaysystem.

The 3D display device is a stereoscopic display system or anautostereoscopic stereo display.

The aspects of the present invention may each be combined with any ofthe other aspects. These and other aspects of the invention will beapparent from and elucidated with reference to the embodiments describedhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described, by way of example only,with reference to the drawings, in which

FIG. 1 shows the relation between the screen parallax p , a displaydistance D between a viewer and a display, an eye distance x_(B) and aperceived distance z_(p) of an object measured from the screen surface,

FIG. 2 shows a comparison of depth ranges typical for Stereo TV, Cinema,and state-of-the-art autostereoscopic displays,

FIG. 3 shows a flowchart of a method according to the present invention,

FIG. 4 shows a parallax map before a transform,

FIG. 5 shows an example of parallax transform for constrained screens,

FIG. 6 shows a parallax map after transform,

FIG. 7 shows another example of a parallax mapping,

FIG. 8 depicts graphically an example of an interpolation between twoparallax information transforms to obtain a third parallax informationtransforms matching a target device constraints,

FIG. 9 shows a device according to the present invention, and

FIG. 10 shows a three-dimensional (3D) display device according to thepresent invention comprising said device.

DESCRIPTION OF EMBODIMENTS Introduction to Stereoscopic DepthPerception:

When looking how people perceive depth in the real world, the people seethe world with two eyes, and each eye sees the world from a slightlydifferent position. The brain fuses the images from the left and righteye to get a 3-dimensional impression.

The depth is perceived through different depth ‘cues’, of which somework even by closing one eye (or if a person looks at a photograph or aTV). Those are called monocular depth cues (with the exception ofaccommodation). Depth cues that need both eyes are called binoculardepth cues.

The monocular cues include the perspective, motion parallax, texturegradient, relative size, interposition, shading, depth-of-field andaccommodation. Accommodation means that when the eyes are focused on acertain object, the brain can estimate the distance of that object fromthe tension in the eye muscle that controls the focus. It is the onlyabsolute monocular cue, all others are relative.

The binocular depth cues are convergence and stereopsis. Convergencemeans that the axes of the eyes of a person converge on the object theperson is looking at. If a person looks at an infinitely far object, theeyes are parallel, and when the person tries to look at the tip of itsnose, the eye lines become crossed. Convergence is also an absolute cue,like accommodation. However, accommodation and convergence are onlyminor cues compared to stereopsis. Stereopsis, sometimes referred to as“triangulation”, means the depth “calculation” by “image processing”which the human brain applies based on the disparity between objects onthe retina of the left and right eye.

When considering how stereoscopic depth perception works in a 3D cinemaor on any other stereoscopic or autostereoscopic display, the techniqueis to show a different image to each eye. This can be achieved byencoding the left and right images with different color (anaglyph,Dolby/Infitec), by using polarized light, or by showing left and rightimages sequentially. All these methods require the viewer to wearglasses to filter out the left and right images. The alternative areautostereoscopic displays that don't require the viewer to wear glasses,but show different images to the left and right eye, based on e.g.barriers or lenses on the display. The quality of all methods isdetermined by how good the separation between left and right image is.If some part of the left image is visible in the right eye, too (or viceversa), the resulting effect is called ghosting or cross-talk, and itcan reduce the 3D experience.

Objects that appear to be on the surface of the screen have identicalimages for the left and right eye. Any object floating behind or infront of the screen is shifted slightly in the left/right images. Theshift in the image is usually measured in pixels, and is called‘disparity’. Since that results in a different effect depending on theresolution of the image and the screen size, one only looks at the shiftof the object on the screen surface, which is referred to as ‘screenparallax’. It is defined as a physical distance in order to beindependent from a specific resolution or screen size.

FIG. 1 shows the relation between the screen parallax p, the displaydistance D between the viewer and the display, the eye distance x_(B)and the perceived distance z_(p) of the object, measured from the screensurface. The x-axis represents the display panel and the z-axis thedistance from the display panel.

The relation can be expressed as follows:

$p = {x_{B} \cdot {\left( {1 - \frac{D}{D - z_{p}}} \right).}}$

It is possible to derive the following properties from the equation:

The screen parallax p is 0 when the perceived depth z_(p) is 0.

The screen parallax p is positive when the object appears to be behindthe screen, and negative when the object appears to be in front of thescreen.

For infinitely far objects (z_(p)→∞), p equals the eye distance x_(B).This is the upper bound of p.

An object that floats halfway between the viewer and the screen (with

$\left. {z_{p} = \frac{D}{2}} \right)$

has a parallax of p=−x_(B).

An object that has a distance of D/3 from the viewer has a parallax ofp=−2x_(B) , and object that has a distance of D/4 has a parallax ofp=−3x_(B), and so on.

Sometimes, it is easier to formulate the equation in a different way.Instead of the absolute distance from the screen surface, it is possibleto focus on the relative distance of an object from the viewer (usingthe screen distance D as reference). The relative depth can be expressedas:

$d = {\frac{D - z_{p}}{D}.}$

This measure is sometimes called ‘apparent depth’. If d is 100%, theobject appears to be on the screen surface, if d is 50%, it floatshalfway between the viewer and the screen. If it is greater than 100%,the object appears to be behind the screen. By rewriting the originalequation in terms of d,

${p = {x_{B} \cdot \left( {1 - \frac{1}{d}} \right)}},$

and solving it for d, gives

$d = {\frac{x_{B}}{x_{B} - p}.}$

The above equations are general and apply to all screen sizes, andviewer distances. Unfortunately, is not so easy to create a perfect 3Dexperience on a real screen, because of the following reasons:

the screen size is limited,

there is an accommodation/convergence mismatch,

motion parallax is missing and

The field of view is limited.

To find out why screen size is a problem, one should look at theso-called stereo window, which is the frame around the visible image. Byshifting left and right images horizontally, it is possible to influencewhich objects appear on the screen plane, in front of it, or behind it.Any object behind the screen plane automatically feels natural, almostlike looking through a real window. Problems arise when objects thatfloat in front of the screen are cut off by the screen border. This is acalled window violation. If, for example, an object floats in front ofthe screen and touches the left border on the image for the left eye,parts of the object are cut off in the image for the right eye. Thehuman brain gets conflicting cues, the stereopsis cue tells it that theobject is in front of the screen, but the occlusion cue tells it thatthe object is hidden behind the screen border, and must therefore bebehind the screen. To a lesser extent it also feels unnatural to haveobjects cut off at the top or bottom border.

Currently, only an IMAX screen is wide enough so that persons don't haveto worry about window violations on the left/right borders. On normalcinema screens (approx 10 m width), window violations start to become aproblem and on 3D television sets the problem is inevitable. Looking atabove equations, one can see that to get the same 3D effect in terms ofrelative depth, the physical screen parallax is identical, regardless ofscreen size. To show infinitely far objects in a cinema, the left andright images are shifted by x_(B)=65 mm . This is approximately 0.5% ofthe screen width. To show infinitely far objects in a 3D TV, the leftand right images are shifted by 65 mm as well, but now the shift isalmost 5% of the screen width. To show an object which floats in frontof the screen at a relative distance of 25%, one needs a margin of atleast 1.5% of the width of a cinema screen, but a 15% margin for a TV.Thus, it is a lot harder to have objects hovering in front of peoplesnose with a 3D TV. It only works for objects which are more or less inthe center of the screen.

The conclusion is that smaller screens automatically limit the amount ofdepth that can be shown.

The other major problem is the accommodation/convergence mismatch.Regardless of where the object appears to be, behind or in front of thescreen, one still has to focus the eyes on the screen. Simply put, onlyone of the two absolute depth cues is used in stereoscopic projection,and it contradicts the other one. For an inexperienced 3D audience, therule of thumb is to avoid retinal disparities of more than 1.5 degrees.Anything more than 1.5 degrees leads to eye-strain, and sometimes peopleare unable to fuse both images into one, and will not see any 3D effect.This depends mainly on the quality of the monocular depth cues that helpus to fuse the images with stereopsis. The retinal disparity can becalculated as follows:

$\beta = {{\tan^{- 1}\left( \frac{p}{D} \right)}.}$

Again, the 3D cinema has the advantage here, because the physicalparallax p is small compared to the screen distance D. Assuming a screendistance of D=10 m, then the retinal disparity for infinitely farobjects is only around 0.37 degrees.

Objects that float at a relative distance of 20% have a retinaldisparity of 1.48 degrees, and are about as near as one should go in acinema.

By looking again at the equations for the relative depth, assume aviewer sits in front of a stereo screen. Objects with a relative depthof 50% appear to be halfway between the viewer and the screen. Now, bymoving closer to the screen—the same object still has a relative depthof 50%, but its depth compared to the size on the screen changes. Theobject has less absolute depth. If the viewer moves away from thescreen, the absolute depth is increased. Only in a certain ‘sweet spot’the viewer gets the correct ratio between depth and 2D size. If theviewer sits in that position, the field of view (i.e. angle how largethe screen appears to you) is the same as the field of view of thecamera. This condition is also called ortho-stereopsis, the perfectreproduction of depth that was observed by the camera.

It is impossible to achieve this condition for all viewers. Even for asingle viewer, it means that the whole content has to be created with asingle camera lens and without zoom. Viewers can easily tolerate toolittle depth, since that is what they are used to in 2D TV and 2Dcinema, by they should avoid too much depth which could look unnatural.

How Depth Scaling works for different Screens:

Depth scaling is a process of converting the depth stored in a format tothe target screen's depth range. With the term depth scaling ispreferably meant mapping a disparity/parallax to anotherdisparity/parallax. Various formats such as WOWvx format can show 3D onany screen, for autostereoscopic displays in mobile phones to stereoprojection in cinemas, by always using the full depth capabilities ofeach screen while eye strain and viewing discomfort are reduced tominimum. It should however be noted that the format should not belimited to a certain 3D format, but other formats such as 2D plus Depthfile and interface format could just as well be used.

The aim here is to show content having an appropriate format on anyavailable 3D screen, from hand-held devices to cinema screens. As such,the format should contain a large depth range, so that enoughinformation is available to show it on big screens. As mentionedherinbefore, several factors have to be considered to find the optimaltransform of the original depth information to the target screen.

Starting with the big screens and then go down to hand-held size andinvestigate for each screen size what the optimal configuration is.

Assuming a file is provided which contains a depth range of 25% toinfinity in terms of relative depth. This means the closest objects arefloating in front of the audience at ¼ of the screen distance, and thefarthest objects are at infinite depth. The parallax range goes from 65mm (infinitely far away) to −195 mm (25% relative depth).

Cinema Screens:

On a cinema screen, or to be more precise, on any cinema screen, thereis enough distance between audience and screen, so theaccommodation/convergence discrepancy is not an issue and one does notexpect problems with the stereo window. Consequently, any cinema screencan show the depth range that is encoded in the appropriate format, andthere's no need to transform the depth information. Note that this isnot the same as showing the same stereo images on each screen—thephysical screen parallax stays the same, but this leads to a pixeldisparity that depends on the screen size and pixel resolution. Anexception is IMAX because the field of view is larger than in normalcinemas. To keep the depth aspect ratio of content that was created fora normal cinema, it could be beneficial to move the screen plane awayfrom the audience.

Stereoscopic 3D TV:

Considering a stereoscopic 3D TV with 3D glasses and a screen width oflm (approximately 45″ diagonal), the usual distance between audience andTV set is 3 m.

It is clear that it is not possible to show the whole depth range of theoriginal content, because 25% of relative depth would lead to a retinaldisparity of 3.72 degrees—generally perceived as being too much forcomfortable viewing. Even infinite depth may lead to eye strain if aviewer has to look at it continuously, though it only has a retinaldisparity of 1.24 degrees.

Another problem is that a screen parallax of −195 mm takes up almost 20%of the screen. This would require a margin of at least 20% on both sidesof the object which is supposed to float in front of the screen in orderto not violate the stereo window.

Additionally, if the original content was intended for a cinema screen,then it is likely that watching it on a smaller screen, but with thesame physical screen parallax leads to a feeling of “too much depth”.This is caused by a disproportionate depth aspect ratio, caused by thenow different field-of-view. Objects are smaller in size, but still havethe same depth, e.g. a ball which now appears to have the form of acucumber.

Finally, one should also try to leave the screen plane where it wasintended to be, and not move it too much towards or away from theviewer. The reason is simple: most of the content where a viewer has tofocus on (e.g. small text or other fine details) is best shown on thescreen plane to avoid eye-strain, and content is usually created to putthe objects that the content creator would like the viewer to focus onat that depth. By taking all 4 factors into account, a good depth rangefor the 3D TV could be 25mm to −75 mm (in terms of parallax) and 46.4%to 162.5% (in terms of relative depth). That's of course very subjectiveand just a safe default.

It is interesting to compare this depth range to what a viewer would getif the viewer is shown a stereo movie created for the cinema screen onthe same TV set. Assuming the content is the same as mentioned above,and the movie was made for a 40′ screen size, then the resulting depthrange is 5.3 mm to −16 mm (in terms of screen parallax) and 80% to 109%(in terms of relative depth). With as an example the WOWvx format, thedepth effect can be made up to 4 to 5 times stronger. A diagramcomparing the difference is shown in FIG. 2, which shows a comparison ofdepth ranges of a typical cinema content shown on a cinema screen, thesame stereo (L/R) images shown on a stereoscopic TV, and the samecontent adapted with WOWvx technology (depth scaling and parallaxinformation transforms) shown on the same stereoscopic TV.

Constrained Screens and Smaller Displays:

Contemporary autostereoscopic displays and smaller displays in generalhave a constrained depth and parallax range, simply because the displaysare not large enough to show sufficient parallax without using up asignificant amount of the screen width, or because multiviewautostereoscopic displays have to render several views and need amultiple of the parallax range used by a stereoscopic display of thesame size.

To use this limited depth range to maximum effect, it is possible to useone of the following methods:

Not every shot/scene uses the full depth range, and it is possible tomap the depth range of each shot to the depth range of the display. Itis not possible to get consistent depth over different shots, and thereis no absolute measure of depth, but that is not noticeable on suchscreens.

In shots that do use the full depth range, reducing it to only afraction of the original depth range leads to a card boarding effect,where e.g. faces and other objects appear flat. A good solution is toincrease depth inside of objects at the cost of depth between objects.This can be achieved by embedding a parallax transform in the format.

Objects that are the focus of the scene can be surrounded with a nearand far plane, or a viewing frustum. On a constrained screen, anythingbehind the far plane is projected on the display's far plane andanything in front of the near plane is clipped/projected to the nearplane. This can also be achieved by embedding a parallax transform inthe format.

The viewing conditions for a 3D display device are not always met. Thiswould require that the screen is large and that it must be watched froma large distance and that the separation between the views must be verygood. These viewing conditions are however not always met; hencesometimes an image+depth signal can exhibit too large parallax range ifit was meant for a 3D display device with less restrictive depthcapabilities. It can also be the case that the content was made for adisplay with a limited depth range, which means more depth could bevisualized on a less restrictive display. Simple linear stretching ofdepth can come a long way to increase or decrease the amount of depth,but sometimes a more scene-specific transformation of parallax is calledfor. Such mappings are known in the art, as described in for example“Nick Holliman, Mapping Perceived Depth to Regions of Interest inStereoscopic Images, in Stereoscopic Displays and Applications XV, 2004,available ashttp://www.comp.leeds.ac.uk/edemand/publications/hol04a.pdf”, herebyincorporated by reference.

An example of the use of such a mapping is given in FIGS. 4-6. The leftside of FIG. 4 shows a scene of a road that appears to be behind thescreen and which extends from screen depth to infinity, and the rightside the parallax map. Very close to the viewer a ball is hovering.There is a large gap in depth between the ball and the visible part ofthe road. The parallax range for the whole range may as an example be−65 mm to 65 mm. For screens with constrained depth range, the ballappears very flat when scaling disparity linearly. It would be morepleasant to have the ball use the whole available space in front of thescreen. This can be achieved with a parallax transform as shown in FIG.5, where the x-axis contains the input parallax of the transform, andthe y-axis shows the output parallax. The positive parallax values arescaled linearly (positive parallax is behind the screen, in this casethis is the road). Doing anything else than linear scaling would cause adiscrepancy between monocular and binocular depth cues, and a recreatingother views would show a curved/bent road instead of a straight one. Theparallax range of the ball from −65 mm to e.g. approximately −40 mm isscaled linearly to use the full “space” in front of the constrainedscreen. The gap between the foreground and background object (theparallax range −40 mm to 0 mm) is removed. A parallax mapping as shownin FIG. 5 will accomplish this and result in a modified parallax map(using a smaller parallax scale) as shown in FIG. 6. As will bediscussed here below, the identity transform, i.e. the actual depth datafor display of the image in FIG. 4, along with the supplied transform,the depth data for display of the image in FIG. 6, is used to derive anew one for display range of a specific target device. It should benoted that the parallax transforms are preferably used because different3D displays have different depth range visualization characteristics. Asan example, on smaller screens the depth range is usually smaller thanon a big cinema screen, where one can have objects almost touch theviewer's nose.

Another example of a parallax mapping is shown in FIG. 7, whichidentifies a range of interest between a far and near plane to which thedisplay's depth range is allocated, clipping any depth values outsidethat range.

It should be noted that by receiving such a parallax map as shown inFIG. 4 (the right figure), from this parallax map it is possible toderive said mapping (in this case the identity mapping) from theparallax map, e.g. the parallax transform characterized by (−65 mm, −65mm), (65 mm, 65 mm), instead of the mapping shown in FIG. 5.

As described in the unpublished patent application EP 07116278.8(Attorney Docket PH008849EP1), filed on Sep. 13, 2007, herebyincorporated as whole by reference, it is advantageous to send parallaxtransforms describing these mappings along with depth maps so that themapping can be applied (or not) at the receiving end where the viewingconditions and the 3D display properties are known. This way, thecontent can be viewed on displays with a variety of parallax rangecapabilities because the parallax maps can still be adapted to the 3Ddisplay and viewing conditions. So the image and depth from FIG. 4 couldbe accompanied by information describing the parallax transform shown inFIG. 5, or conversely, if the depth map from FIG. 6 would be encoded inthe content, the inverse of the transform shown in FIG. 5 could be sentalong as meta-data to allow reconstruction of the depth map shown inFIG. 4.

Furthermore, the original depth signal (or rather the meta-data like theparallax scale and offset in MPEG-C part 3) gives rise to one parallaxrange, whereas applying the provided parallax transform as described inPH008849EP1 results in a second depth signal with a second parallaxrange.

In one embodiment, the aim of present invention is to deal with thescenario where neither of the said parallax ranges, i.e. neither theparallax range of the original depth signal nor second parallax rangematches parallax range of a target 3D display. In such a situation, anovel transform can be computed from the two transforms (e.g. theprovided transform and the identity transform) by interpolation (or ifneed be extrapolation). In this way, the depth range of the signal canbe tuned to the available parallax range of a 3D display.

FIG. 3 shows a flowchart of a method according to the present inventionof processing parallax information comprised in a signal.

In step (S1) 301, a signal is received comprising a parallax mapassociated with further image information.

In step (S2) 303, first data is obtained from the signal indicative offirst parallax map constraints.

The data indicative of the parallax map may be either parallaxinformation (a.k.a disparity information, indicating the (horizontal)amount of displacement between views), or distance information (dataindicative how far in front or behind of the display the scene at thatlocation is positioned). The parallax map constraints may as an exampleinclude the display distance between a viewer and the 3D display device,or a location parameter indicating the position of the viewer from the3D display device, a combination thereof.

In step (S3) 305, second data is obtained from the signal indicative ofsecond parallax map constraints.

In step (S4) 307, third data matching third parallax map constraints ofa target device is determined by means of processing at least the firstdata and the second data. This third data is adapted to generate anupdated signal matching the parallax map information constraints of thetarget device.

In one embodiment, the first data and the second data is parallaxinformation transforms and the first and the second parallax mapconstraints is first and second depth ranges for the image information.

The term with parallax information may include depth-related informationor disparity-related information or a combination of both.

In this embodiment, the third data is third parallax informationtransform and the third parallax map constraint is a third depth range.In one embodiment, this first parallax information transform is anidentity information transform of the received parallax map, i.e. theactual depth data for display of the image, and the at least secondparallax information transform is parallax information transformprocessed from the identity information transform (see FIG. 4-6).

As described in PH008849EP1, this at least second parallax informationtransform may be comprised in the video signal as metadata, where themetadata comprises at least one mapping function used in the generationof the parallax information transform, or at least one inverse of amapping function used in the generation of the parallax informationtransform, or at least one coefficient for a mapping function used inthe generation of the parallax information transform, or a combinationthereof.

In one embodiment, the step of determining third data matching the thirdparallax map constraints of the target device comprises interpolatingbetween two respective parallax transforms from a set of parallaxinformation transforms, the set of parallax transforms comprising thefirst and second parallax transforms, the third data being a thirdparallax information transform matching the depth range of the targetdevice. The two or more parallax information transforms may be selectedfrom the second parallax information transforms or from the identityinformation transform of the received parallax map and one or more ofthe second parallax information transforms. In another embodiment, theinterpolation comprises interpolating between two (or more) depth mapswhich each have their range.

Accordingly, if more than one parallax transform is provided,higher-order interpolation can be used, or a suitable subset of theavailable transforms chosen to perform interpolation. One commonimplicit third transform is the linear transform which linearlycompresses or expands the content parallax range to the display parallaxrange.

The selection of which two or more parallax information transforms usedas input in determining the updated parallax information transform maybe selected based on a selection rule. This selection may be based onselecting those parallax information transforms that fall within apre-determined depth range of the target device. As an example, theselection of which two or more transforms to use could be based onselecting those parallax information transforms which range lie closestto the target device range, preferably one having smaller and one havinglarger range.

As an example, suppose the parallax range of the original depth data is[0 . . . 12], and suppose a parallax transform is available which mapsthis range to [0 . . . 4], maybe by selecting the sub-range [4 . . . 8]from the 0 to 12 and clipping values below 4 and above 8. This transformcan be characterized by the mappings of 0−>0, 4−>0, 8−>4 and 12−>4 (withlinear interpolation between them). If the target device is a displaydevice which has a depth range of [0 . . . 8], it is possible to computea new parallax transform by interpolating between the identity transformand the supplied parallax transform. Since 8 is the average of 12 and 4,this is accomplished by averaging the mappings. The result is a mapping:0−>(0+0)/2, 4−>(0+4)/2, 8−>(4+8)/2, 12−>(4+12)/2. This new transform canthen be applied to the parallax data instead of the supplied transform(so for example a parallax of 2 would map to 1, since 0 maps to 0 and 4maps to 2). This is depicted graphically in FIG. 8, where the “diamonds”indicate the parallax range of the original depth data [0 . . . 12], the“triangle” the parallax range of the supplied parallax transform whichmaps this range to [0 . . . 4] and the triangles are the third and newparallax transform [0 . . . 8]. This is an example of how a noveltransform can be computed from two transforms by interpolation. In thisway, the depth range of the signal can precisely be tuned to theavailable parallax range of a 3D target display.

The interpolation may also be done using parallax transforms which have“control points” at different locations. E.g. referring the previousexample, the supplied parallax transform in the example has controlpoints at 0, 4, 8 and 12. If there were another parallax transform 0−>0,6−>8, 12−>12 (an extra control point at (6,8) compared to the identitytransform), interpolation would have to compute for this parallaxtransform what the values are at 4 and 8, and for the former parallaxtransform what the value is at 6, and then a new parallax transformwould be created with control points at 0, 4, 6, 8, and 12.

Continuing with the flowchart in FIG. 3, in step (S5) 309 the updatedsignal is subsequently forwarded to the target device where the updatedsignal is used to adjust the parallax map so as to render image elementsfor view information for a three-dimensional image, that falls withinthe available parallax range of the target device. This step disclosesthe scenario where the above mentioned processing steps are performedexternally from the target device, i.e. the above mentioned processingsteps do not necessarily be performed by the target device, but with anyreceiver device (not the target device). Such a device may be aprocessing unit which allows the processor to transform the data byinterpolating in order to derive a signal suitable for requirementspresented by the target device, e.g. the 3D display device. As a resultthe processing device could be in a set-top box (e.g. when the incomingsignal already has two relevant signals associated), it could be in a 3Ddisplay device, it could be in a “display adapter” that convertsinformation in memory to video output to a display, or a program runningon a PC computer.

FIG. 9 shows a device 900 for processing parallax information comprisedin a signal 901, where the device 900 comprises a receiver (R) 910 and aprocessor (P) 911. The receiver is adapted for receiving a signalcomprising a parallax map associated to image information 902. Theprocessor (P) 911 is adapted for obtaining first data from the signalindicative of first parallax map constraints, obtaining second data fromthe signal indicative of second parallax map constraints, anddetermining third data matching third parallax map constraints of atarget device 904 by means of processing at least the first and thesecond data. This third data is adapted to generate an updated signalmatching the parallax map information constraints of the target device904. The processing steps performed by the processor (P) 911 havealready been discussed in the flowchart in FIG. 3.

The device may be an integral part of a set-top box 905, Blu-ray Discplayer 905, a 3D display device 1000, a stereo display, a PC computerdevice 905, a portable computer device, and the like.

As depicted in FIG. 9 and as discussed previously in FIG. 3, theprocessing may be performed at the display device side (i.e. at theend-receiver side), or externally where the third data is subsequentlytransmitted by a transmitter (T) 906 to the target device 904 via awired or wireless communication channel 907, which is provided with areceiver 903 to receive the process the information so as to generatethe updated signal matching the parallax map information constraints ofthe target device 904. In case of the externally processing, the displaydevice 900 might be adapted to read in e.g. a .wowvx file with aparallax transform in it, and set the output mode for compositor to astereo mode for a certain display (which has a certain range). In thatway the compositor would render a left and right picture using theadjusted parallax.

As an example, a Blu-ray disc may contain video+parallax information fora certain 3D depth range and a parallax transform which allows mappingthe parallax information to a new parallax map which can be used for asecond display type with a different depth range. The Blu-ray discplayer can play this disc, convert the information from the compressedformat to a display format and send all this information via for exampleHDMI to a 3D display device. Either the display device or the Blu-raydisc player can apply the method disclosed herein to compute a novelparallax transform which can be used to map the parallax information tothe display range of the 3D display in question (the Blu-ray disc playercould ascertain the display's display range from for example the EDIDinformation). If the Blu-ray disc player would implement the method, itwould replace the parallax transform read from the disc with the noveltransform which fits the 3D display device when communicating the videodata to the display. Alternatively, the Blu-ray player could apply thenovel parallax transform to the parallax maps read from the disc, andsend the novel parallax maps to the display instead of the ones readfrom disc. There would then be no need anymore then to send a parallaxtransform anymore would provide compatibility for a 3D display whichdoes not implement the method disclosed herein. Alternatively, theoriginal parallax transform from the disc is sent to the 3D display andthe 3D display carries out the method disclosed herein to compute anovel parallax transform.

FIG. 10 shows a three-dimensional (3D) display device 1000 comprisingthe device 900 from FIG. 9. This 3D display device may be anauto-stereoscopic display system, a stereoscopic display system or astereo display.

The method according to the present invention may be implementedadvantageously on a wide variety of processing platforms.Implementations can be envisaged that operate on a general purposecomputer, digital signal processor or another programmable processor.Alternatively the invention may be implemented in a pre-programmedhardware implementation comprised on an Application Specific IntegratedCircuit.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive; theinvention is not limited to the disclosed embodiments. It is furthernoted as described above with reference to FIG. 10 that the inventionmay be embodied in a product such as a display, a set top-box, or otherdevice. In the latter case the invention may be incorporated in,implemented on processing platforms targeted at this very purpose and/ormore general purpose processing platforms.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. A single processor or other unit may fulfill thefunctions of several items recited in the claims. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measured cannot be used toadvantage. A computer program may be stored/distributed on a suitablemedium, such as an optical storage medium or a solid-state mediumsupplied together with or as part of other hardware, but may also bedistributed in other forms, such as via the Internet or other wired orwireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope.

1. A method of processing parallax information comprised in a signal,comprising: receiving a signal comprising at least one parallax mapassociated with further image information (301), obtaining first datafrom the signal indicative of first parallax map constraints (303),obtaining a second data from the signal indicative of second parallaxmap constraints (305), determining third data matching third parallaxmap constraints of a target device by means of processing at least thefirst data and the second data (307), the third data being adapted togenerate an updated signal matching the parallax map informationconstraints of the target device.
 2. A method according to claim 1,wherein the first data and the second data are parallax informationtransforms, the first and the second parallax map constraints beingfirst and second depth ranges for the image information, the third databeing a third parallax information transform and the third parallax mapconstraints being a third depth range.
 3. A method according to claim 2,wherein the first parallax information transform is an identityinformation transform of the received parallax map.
 4. A methodaccording to claim 3, wherein the second parallax information transformis parallax information transform obtained by using the identityinformation transform as input which is processed, the processingresulting in outputting the second parallax information transform.
 5. Amethod according to claim 1, wherein the signal is a video signal andwhere the second parallax information transform is comprised in thevideo signal as metadata.
 6. A method according to claim 5, wherein themetadata comprises at least one of: a mapping function related to theparallax information, an inverse of a mapping function related to theparallax information, and a coefficient for a mapping function relatedto the parallax information.
 7. A method according to claim 2, whereinthe step of determining third data matching the third parallax mapconstraints of the target device comprises interpolating between tworespective parallax information transforms from a set of parallaxinformation transforms, the set of parallax information transformscomprising the first and second parallax information transforms, thethird data being a third parallax information transform matching thedepth range of the target device.
 8. A method according to claim 7,wherein the set of parallax transforms further comprises a furtherparallax transform based on further data from the signal.
 9. A methodaccording to claim 7, wherein the parallax information transforms usedas input in determining the updated parallax information transform areselected based on a selection rule.
 10. A method according to claim 9,wherein the selection rule defines selecting parallax informationtransforms that fall within a pre-determined depth range of the targetdevice.
 11. A method according to claim 1, wherein the target device isa 3 dimensional (3D) display system and where the respective parallaxmap constraints comprise at least one from: the parallax or depth rangeof the 3D display device, the display distance between a viewer and the3D display device, and a location parameter indicating the position ofthe viewer from the 3D display device.
 12. A method according to claim2, wherein the updated signal is subsequently forwarded to the targetdevice where the updated signal is used to adjust the parallax map so asto render image elements for view information for a three-dimensionalimage that falls within the available parallax range of the targetdevice.
 13. A computer program product for instructing a processing unitto execute the method step of claim 1 when the product is run on acomputer.
 14. A device (900) for processing parallax informationcomprised in a signal (901), comprising: a receiver (910) for receivinga signal comprising at least one parallax map associated to imageinformation, a processor (911) for obtaining first data from the signalindicative of first parallax map constraints (301), obtaining seconddata from the signal indicative of second parallax map constraints (303)and determining third data matching third parallax map constraints of atarget device (904, 100) by means of processing at least the first andthe second data, the third data being adapted to generate an updatedsignal matching the parallax map information constraints of the targetdevice (904, 1000).
 15. A three-dimensional (3D) display device (1000)comprising a device as claimed in claim 14.